Job Description :
Design, develop, and maintain ETL pipelines using Databricks, PySpark, and ADF to extract, transform, and load data from various sources.
Must have good skills in Pyspark, Programming, code remediation, etc.,
Must have good working experience on Delta tables, deduplication, merging with terabytes of data set
Optimize and fine-tune existing ETL workflows for performance and scalability. 2 to 3 years of experience in ADF is desirable (Medium expertise required)
Must have experience working with a large data set
Proficient in SQL and must work on complex joins, Subqueries, functions, procedure
One should be self-driven and work independently without support
